A Scalable HTTP Server: The NCSA Prototype

While the World Wide Web (www) may appear to be intrinsically scalable through the distribution of files across a series of decentralized servers, there are instances where this form of load distribution is both costly and resource intensive. In such cases it may be necessary to administer a centrally located and managed http server. Given the exponential growth of the internet in general, and www in particular, it is increasingly more difficult for persons and organizations to properly anticipate their future http server needs, both in human resources and hardware requirements. It is the purpose of this paper to outline the methodology used at the National Center for Supercomputing Applications in building a scalable World Wide Web server. The implementation described in the following pages allows for dynamic scalability by rotating through a pool of http servers that are alternately mapped to the hostname alias of the www server. The key components of this configuration include: » Cluster of identically configured http servers » Use of Round-Robin DNS for distributing http requests across the cluster » Use of distributed File System mechanism for maintaining a synchronized set of documents across the cluster. » Method for administering the cluster. The result of this design is that we are able to add any number of servers to the available pool, dynamically increasing the load capacity of the virtual server. Implementation of this concept has eliminated perceived and real vulnerabilities in our single-server model that had negatively impacted our user community. This particular design has also eliminated the single point of failure inherent in our single-server configuration, increasing the likelihood for continued and sustained availability. While the load is currently distributed in an unpredictable and, at times, deleterious manner, early implementation and maintenance of this configuration have proven promising and effective. Please note that it is neither the intention of the author nor of NCSA to either endorse or malign any particular architecture or hardware type. For this reason I will make no specific references to architecture types or equipment performance. Note also that any references to IP addresses or host names are not representative of the actual information and the reader should not attempt to make connections using this information.